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The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to 
breaking the link between family income and educational achievement, ensuring that children from all 
backgrounds can fulfil their potential and make the most of their talents. 


The EEF aims to raise the attainment of children facing disadvantage by: 


e Identifying promising educational innovations that address the needs of disadvantaged 
children in primary and secondary schools in England; 


e Evaluating these innovations to extend and secure the evidence on what works and can be 
made to work at scale; 


e Encouraging schools, government, charities, and others to apply evidence and adopt 
innovations found to be effective. 


The EEF was established in 2011 by the Sutton Trust, as lead charity in partnership with Impetus 
Trust (now part of Impetus-The Private Equity Foundation) and received a founding £125m grant from 
the Department for Education. 


Together, the EEF and Sutton Trust are the government-designated What Works Centre for improving 
education outcomes for school-aged children. 
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About the evaluator 


The project was independently evaluated by a team from the Institute for Effective Education (IEE), 
University of York 


The lead evaluator was Dr Mary Sheard. Professor Bette Chambers took over responsibility upon the 
death of Mary Sheard in summer 2014. 


Contact details: 


Professor Bette Chambers 

Director, Institute for Effective Education 
University of York 

Heslington, UK 

YO10 5DD 


p: 01904 328153 
e: bette.chambers@york.ac.uk 
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Executive summary 


The project 


Units of Sound (UofS) is a computer-based programme designed to help struggling readers with their 
reading and spelling skills. It is a structured, multisensory programme that covers reading and spelling 
from simple phonics skills through to adult reading levels. It involves a high level of independent work 
by the student, with small groups of students supervised by a teacher or teaching assistant. Each ‘unit 
of sound’ (or phonic code) is introduced separately, then used in words, and then sentences. 


This evaluation assessed the reading ability of 786 Year 7 students in 45 schools. Students were 
randomly assigned within the schools to receive the intervention if they had scored below Level 4 on 
their Key Stage 2 SATs. 


The evaluation was funded by the Education Endowment Foundation as one of 23 projects focused on 
literacy catch-up at the primary-secondary transition for students who did not achieve Level 4 in 
English by the end of Key Stage 2. 


Two staff members in each school were identified to implement the interventions with groups of 
approximately five Year 7 students at a time. The designated teachers/teaching assistants in the 
participating schools undertook the 10-hour initial online training in UofS. Of the 45 schools that took 
part in the study, 31 had four sessions of additional face-to-face training and support from Dyslexia 
Action (DA) coaches and 14 had only online distance training and support instead of face-to-face 
support. 


Students spent one 60-minute session with the teaching assistant present and a further 30-minute 
session on another day working independently on the material covered with the teaching assistant. 
Schools varied in how they allocated the time. The intervention lasted for 18 weeks. 


Students were evaluated at baseline on the New Group Reading Test (NGRT) and then at follow-up 
on the NGRT and the Single Word Spelling Test (SWST). A light-touch process evaluation was 
conducted to determine teaching assistants’ and students’ perceptions of the programme. This 
included surveys of teaching staff and students and interviews with the DA coaches. 


Key conclusions 


1. This attempt to evaluate the UofS programme was severely compromised and no firm 
conclusions can be drawn from it. 


2. Schools need dedicated staff and better ICT support in their school if they are to implement 
UofS. 


3. Students in schools that participated in the distance-learning-only support seemed to perform 
similarly to those who received face-to-face support. 


4. More research with better recruitment, better technology support, and lower attrition would be 
needed to determine the effectiveness of the programme and the generalisability of the 
effects. 
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What impact did it have? 


This attempt to evaluate the UofS programme was severely compromised by high numbers of schools 
dropping out and students not completing testing. Consequently, no firm conclusions can be drawn 
from it. 


From the collected data, neither outcome (reading or spelling ability) showed a significant impact of 
the intervention on students compared to control students’ achievement. Overall, there were negative 
effects that were not statistically significant, increasing the likelihood that they may have occurred by 
chance. The results for the children eligible for free school meals (FSM) were considerably more 
negative, but still not significant. 


How secure are these findings? 


The evaluation was set up as an efficacy trial, meaning that it seeks to test evaluations in the best 
possible conditions to see if they hold promise. They do not indicate the extent to which the 
intervention will be effective in all schools. 


Overall, the findings from this evaluation are judged to be of weak security with one padlock. The very 
high attrition rate of 46% was mostly at the school level, but because the study used within-school 
randomisation there was no evidence that the attrition biased the findings. However, the high attrition, 
technology challenges, and other issues stated above, make it impossible to generalise the findings to 
other schools. 


In this randomised controlled trial, the online New Group Reading Test (NGRT) was administered to 
the participating students before the intervention began in each phase and post-tested following 
implementation of the intervention in each phase, along with the online Single Word Spelling Test 
(SWST). 


The surveys indicated that the teachers, students and teaching assistants who experienced the 
programme had mosily positive perceptions of the programme and believed that it had improved 
learning. However, surveys and interviews with the DA coaches suggested that a lack of technology 
support and engagement with the online training may have decreased implementation fidelity. 


How much does it cost? 


Owing to variation in the number of students per school and the lack of data about how many teachers 
and how many teaching assistants participated in the study, it is impossible to determine the exact 
cost or cost-effectiveness of the intervention as implemented in this study. If a school were purchasing 
the UofS programme, DA estimates that it would cost about £2,500 per school for the resources and 
training. If there were ten students in a school, that would be £250 per student. 


Effect size Estimated 
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Note: None of the differences were statistical - significant, sugges ting that any differences occurred by chance. 

*For more information about evidence ratings, see Appendix 8 in the main evaluation report. Evidence ratings are not provided 
for sub-group analyses, which will always be less secure than overall findings. 

**For more information about cost ratings, see Appendix 9 in the main evaluation report. 
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Introduction 


Dyslexia Action (DA) is a national charity that offers help and support to individuals affected by 
dyslexia and other learning difficulties. Their head office is in Egham, Surrey and they have a network 
of 23 regional centres and 97 teaching locations throughout the UK. The Dyslexia Institute was 
established in 1972. In 2005, the Institute merged with the Hornsby International Dyslexia Centre, and 
they changed their name to Dyslexia Action. DA provides a number of interventions for students with 
learning difficulties, including Units of Sound (UofS), the programme evaluated in this study. 


Intervention 


UofS is a computer-based programme designed to help struggling readers with their reading and 
spelling skills. It covers reading and spelling from simple phonics skills through to adult reading levels. 
It is a structured, cumulative and multisensory programme that involves mostly independent work by 
the student, with about five students at a time supervised by a teacher or teaching assistant. Each 
‘unit of sound’ (or phonic code) is introduced separately, before being used in words, then sentences. 
See www.unitsofsound.net for more information on the materials. 


In the first week of the intervention, students completed the UofS placement assessment and this 
determined at which level each student should begin the programme. Then students were to 
participate in the intervention for at least 60 minutes per week for 17 weeks. It was recommended that 
they have 60 minutes with the teaching assistant and another 30 minutes of independent work on the 
programme. We don't have any data on how many sessions students actually participated in. 


In this project, the software and hardware were installed on the schools’ computers and activated for 
training to begin. The designated teachers/teaching assistants in the participating schools undertook 
the 10-hour initial online training in UofS. Most of the schools received four sessions of additional face- 
to-face training and support from DA coaches. Participating teachers/teaching assistants in about one 
third of the schools had additional online training and support instead of face-to-face support. This was 
done in some cases because the schools were remote or because there were too many schools in 
one DA area for the DA staff to serve. 


Background evidence 


The Department for Education (DfE) is concerned about the challenges that students face in the 
transition from primary to secondary school, especially those students who fail to reach Level 4 on 
their KS2 SATs English tests (The Rose Review, June 2009). If students cannot read fluently with 
good comprehension when they begin secondary school, their chances of completing school and 
going on to further or higher education are very slim. Children from poorer backgrounds are less 
successful than their more advantaged peers in tests across a range of subjects, in all 30 OECD 
countries (Ainscow et al, 2010). 


An interim review from the EEF" suggests that there are approaches that can help struggling readers, 
even in early secondary school. Some of the characteristics of UofS indicate that it contains some 
features that should lead to its effectiveness, such as one-to-one tuition, and that the intervention is 
targeted to the individual student’s current reading level (Slavin, Lake, Davis and Madden, 2011). 
Based on this prior evidence, the DfE contracted the EEF to organise evaluations of interventions to 
help struggling readers with the transition to secondary school. DA’s UofS was chosen as one of the 
promising programmes to evaluate. 


In terms of evidence for UofS, a University of Durham evaluation by the Centre for Evaluation and 
Monitoring within 35 schools provided some evidence that DA’s Partnership for Literacy (P4L) 


: http://educationendowmentfoundation.org.uk/uploads/pdf/Transition_Catch_Up_Literature_Review.pdf 
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programme can lead to improved literacy. The UofS software intervention (integral to P4L) indicated 
that students improved in reading after a ten-week intervention (Dyslexia Action, 2012) but there was 
no control group so we cannot be sure what level of improvement it made. Anecdotal and qualitative 
evidence suggests that implementing UofS at the transition stage could provide the boost that 
struggling readers in Year 7 might need to help them succeed in English in secondary school. 


This preliminary evidence suggested that UofS has demonstrated sufficient promise to warrant 
conducting an efficacy trial but not strong enough evidence to conduct a larger scale effectiveness 
evaluation. The programme had previously been used with younger children in primary school so we 
do not know how effective it will be with secondary school students. 


Evaluation objectives 


The primary objective of the evaluation was to estimate the impact of UofS on the reading and spelling 
ability of struggling Year 7 readers in 50 secondary schools. 


The secondary evaluation objectives were: 


e To assess the differential impact of providing face-to-face support or distance-learning-only 
support for schools to implement the programme. 


e To conduct subgroup analyses to determine the effects of the programme for children eligible 
for free school meals. 


e Toassess teaching assistants’ perceptions of the UofS programme, whether they thought it 
improved their students’ reading and spelling abilities, the conditions they believed were 
necessary for success and the barriers to implementation. 


e To assess students’ perceptions of the UofS programme and its impact on their reading and 
spelling abilities. 


Evaluation team 


Mary Sheard, PhD. Dr Mary Sheard was a Research Fellow at the Institute for Effective Education 
with interests in teacher learning and professional development; student learning with technology; and 
learning with representations. Unfortunately, Dr Sheard was ill for the latter part of the study and died 
in summer, 2014. 

Bette Chambers, PhD. Professor Bette Chambers is Director of the Institute for Effective Education at 
the University of York. Professor Chambers develops and evaluates effective practices in early 
childhood and literacy education and promotes the use of evidence-based practices. 


Louise Elliott. Louise Elliott is the Data Manager at the Institute for Effective Education, where she 
manages all database organisation, data entry, cleaning, and the statistical analyses conducted in the 
research work. 


Ethical review 


We obtained ethical approval from the Department of Education, University of York Ethical Review 
Panel, which normally approves our research projects. See Appendices 3 and 4 for a copy of the 
agreement form that head teachers signed committing their schools to participating in the project, 
letters of consent from the DA coaches, and information and opt-out consent forms for parents. 


Data were managed in accordance with the Data Protection Act (1998). The trial database is securely 
held and maintained on the University of York’s research data protection server, with non-identifiable 
data. Confidentiality is maintained and no one outside the trial team has access to the database. Data 
were checked for missing data and/or double entries. All outputs were anonymised so that no schools 
or students can be identified in any report or dissemination of results. 
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Methodology 


Trial design 


This efficacy evaluation was an individual level randomised multi-site trial with Year 7 students who 
were identified as struggling readers. The previous evidence for the programme did not yet warrant a 
large-scale effectiveness trial. This was a delayed-treatment trial with the students in the control 
condition participating in the UofS programme after the post-test data collection. 


The independent variable for the impact analyses was allocation or not to the intervention. In both 
Phase 1 and Phase 2, it was planned that DA would recruit approximately 25 secondary schools in 
areas of multiple deprivation across England for a total of 50 schools. The IEE research team was to 
randomly select on average 20 students from each school who had achieved below Level 4 on the 
Key Stage 2 SATs in reading, and then randomly from those that met the criteria. The online (New 
Group Reading Test (NGRT) was to be administered to the selected students, and then they would be 
randomly assigned by the IEE to receive the intervention programme either during the study or as a 
delayed treatment at the end of the evaluation. 


The dependent variables were the students’ scores on the New Group Reading Tests (NGRT) and 
the Single Word Spelling Test (SWST) at post-test, with the NGRT pre-test score as a covariate. We 
were to conduct both intent-to-treat analyses and analyses of the scores of students who participated 
in the intervention. 


Ten of the schools were to receive only online training and support, while the rest were to receive a 
combination of online training and in-person follow-up support. Dyslexia Action decided which schools 
were to be the online-only schools because there were too many schools in an area for DA staff to 
serve face-to-face or because of the remote location of the schools. As it turned out three schools in 
Phase 1 and eight in Phase 2 had the distance learning and support. 


Additional analyses were conducted to determine the differential effects for online-only versus online 
plus face-to-face training and support. Subgroup analyses were conducted to determine differential 
effects for students eligible for free school meals. 


Owing to recruitment problems, high attrition, and missing data for implementation fidelity and 
students’ ability levels, the planned analyses for these factors were not conducted. 


Eligibility 


DA chose 12 of its regional centres to take part in the project. They approached all the secondary 
schools located in those regions to participate in the evaluation. The original target of 50 schools 
proved to be a challenge and the EEF allowed DA to run the project in two phases, one beginning in 
February 2013 and the other in October 2013. In Phase 1 they signed up 24 schools and in Phase 2 
they recruited a further 21 schools, making a total of 45 schools. Two staff members in each school 
were identified to implement the interventions with groups of approximately five students at a time. 
Head teachers signed agreements to participate in the study and parents signed opt-out forms before 
randomisation if they did not want their child to participate. 


Each school identified the Year 7 students who had scored below Level 4 on their Key Stage 2 
reading SATs and sent those names to the IEE research team. In the few schools that had more than 
20 struggling readers, the IEE data manager used a random number generator to randomly select 20 
of those students to participate in the study. However, in most schools there were fewer than 20 
students who scored below Level 4, so half of them were randomly assigned to participate in the UofS 
programme and half to receive the support that the school regularly provided for struggling readers. 
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Intervention 


UofS is a computer-based programme designed to help struggling readers with their reading and 
spelling skills. It covers reading and spelling from simple phonics skills through to adult reading levels. 
It is a structured, cumulative and multisensory programme. It involves a high level of independent 
work by the student, with about five students at a time supervised by a teacher or teaching assistant. 
Each ‘unit of sound’ (or phonic code) is introduced separately, before being used in words, then 
sentences. See www.unitsofsound.net for more information on the materials. 


In this project, the software and hardware were installed on the schools’ computers and activated for 
training to begin. The designated teachers/teaching assistants undertook the 10-hour initial online 
training in UofS. The number and composition of the teams implementing the UofS programme in 
each school varied, in part owing to the number of students involved in the project. Generally, two 
individuals who had had responsibility for supporting struggling readers were chosen to implement the 
UofS programme. Sometimes this was teaching assistants or a combination of a teaching assistant 
supervised by a teacher who also did some of the implementation. We do not know how many were 
teaching assistants and how many were teachers but most were teaching assistants so that is how 
we will refer to them in this report. 


In 31 of the schools, trained specialist staff from the 12 DA centres around England provided the 
teaching assistants implementing the programme with four days of additional face-to-face training and 
support. Participating teachers/teaching assistants in 14 schools had additional online training and 
support instead of face-to-face support. The schools that received online support only, did so because 
either they were in remote areas or there were too few DA coaches to support all the participating 
schools in that area. 


Each intervention student was supposed to participate in UofS for at least 60 minutes each week for 
18 weeks. In the first week, the Units of Sound Placement Test was administered to the students in 
the intervention groups at the start of programme implementation to assess their current level of 
reading ability on the UofS programme. Those data was used to start the students on the appropriate 
level of the programme. 


It was recommended that students spend one 60-minute session with the teaching assistant present 
and a further 30-minute session on another day. Schools varied in how this time was allocated. In 
some schools students were taken out of class to participate and in others they participated during 
study periods. Sessions were supposed to be conducted in quiet, private work areas where students 
would feel comfortable to make (oral) responses without being overlooked or overheard by peers. 
Schools were instructed to provide their regular services to the students in the control condition. 
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Outcomes 


Primary outcome: New Group Reading Test (NGRT) 


Students completed the online NGRT (produced by GL Assessment) at pre-test (NGRTA) and post- 
test (NGRTB), which was highly correlated with Key Stage 2 results. (See http://www.gl- 
assessment.co.uk/products/new-group-reading-test-digital for details.) 


The NGRT is a standardised reading test that has three sections — phonics, sentence completion, and 
passage comprehension — which can be combined into a composite reading score. Students begin 
with the sentence completion section and depending on their score, progress to either phonics tasks 
or a passage comprehension test. The difficulty is increased until the questions become too 
challenging. 


This test was selected so that there would be comparable data among all the EEF transition projects 
and because it assessed the skills that the programme targeted. 


Both the pre-tests and the post-tests were effectively blinded as the tests were in digital form and 
were marked electronically by GL. For this reason, there were no plans to invigilate and tests were 
delivered by schools. 


Secondary outcome: Single Word Spelling Test 


As an additional post-test measure, students completed the Single Word Spelling Test (digital) 
(SWST) produced by GL Assessment, which produces students’ results by standardised age scores, 
spelling ages, percentile ranks, and analysis by word level against national scores. This test was 
administered because it was believed by the developers that UofS might improve children’s spelling. 


Sample size 


The original design was to have 50 schools in one cohort, with an average of 20 students within each 
school, with randomisation at the student level. We assumed a pre-post correlation of 0.75 based on 
previous use of these reading tests. Figure 1 illustrates that a sample size of 1,000 (500 per condition) 
would make it possible to detect an effect size of 0.12 (a small effect). 
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Figure 1. Planned sample size of 50 schools with 20 students each = 1000 students 
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In Phase 1, the timeframe for getting schools recruited and up and running was short — November to 
January 2012/13, over the Christmas period. As a result, in Phase 1 Dyslexia Action ended up with 
only 24 schools. Thus a second phase was conducted in autumn 2013. 


In Phase 2, DA ended up with 21 schools that had students participating. This made a total of 786 
students across both phases allocated to condition. Figure 2 illustrates that a sample size of 786 (393 
per condition) would still make it possible to detect an effect size of 0.14. 
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Figure 2. Sample size at randomisation with 45 schools = 786 students 


- a = 0.050 


N=786,R2,= 0.51 


09 


08 


07 


06 


0.5 


cai: i= ee 8 


04 
0.3 
02 


0.1 


0.06 0.12 0.18 0.24 0.30 
Unconditional effect size 


The MDES for the sample that was analysed is 0.19, using the actual pre-post correlation of 0.714. 
See Figure 3 below. 


Figure 3. MDES for NGRT final sample size of 427 
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Randomisation 


Twenty students per school, randomly allocated to condition, would mean that ten students would 
participate in the intervention. Some schools had fewer than 20 struggling readers, so they were all 
randomised to condition. Some had too few staff to serve 10 students so we randomly selected twice 
as many as they could serve and then randomly assigned them to condition. Where a school had 
more than 20 struggling readers (10 out of 45 schools), the researchers randomly selected 20 of 
those who performed below Level 4 on their Key Stage SATs reading and then randomly allocated 
those individuals to UofS or control conditions. In 35 of the 45 schools (78%), all of the students who 
achieved below Level 4 on the Key Stage 2 SATs in reading in Year 7 were individually randomly 
assigned to condition using a random number generator run by the IEE data manager. 


Randomisation increased the likelihood that the implementation and control students would be similar. 
Year 7 students in the intervention and control groups were post-tested in the summer term 2012/13 
in Phase 1 and at the end of the spring term 2013/14 in Phase 2. 


Analysis 


We had intended to conduct intent-to-treat analyses; however, even with many requests for the data 
from the IEE researcher and from the DA coaches, no schools who had stopped implementing the 
programme provided post-test data, making such analyses impossible. Thus we analysed data only 
for the schools that did not drop out of the study. 


Because there were two phases of implementation, we conducted an initial T-test on the pre-test 
scores to determine if the students in each phase were equivalent. There were no differences 
between the pre-test scores so we combined the phases for the analyses presented here. 


A linear regression was conducted on the NGRT and SWST post-test scores, using the NGRT pre- 
test scores as the covariate with treatment condition as the independent variable, including phase as 
a fixed factor and school as a random factor to account for cluster effects. We used Hedges’ g to 
calculate effect sizes on NGRT and SWST outcomes. 


A subgroup analysis of students eligible for FSM was conducted with data that we received from the 
National Pupil Database (NPD). Also, a regression analysis was conducted to compare the post-test 
scores of the students in treatment condition in the schools that received the UofS programme with 
face-to-face follow-up support, to those in schools that received only the distance-learning treatment, 
and the control group. 


We intended to conduct analyses to determine differential effects for students who experienced 
different levels of implementation fidelity and different levels of teacher education and experience, but 
we did not receive that data and could not conduct these analyses. 


Process evaluation methodology 


A light-touch process evaluation was conducted to examine implementation fidelity, teacher and 
student perceptions, and contextual factors. The process evaluation consisted of three elements 
completed towards the end of the evaluation: 


a) semi-structured interviews with the DA coaches who supported the implementation of the 
programme 

b) paper-and-pencil teaching assistant/teacher surveys 

Cc) paper-and-pencil student surveys. 


Education Endowment Foundation 13 


Units of Sound 


DA coach interviews 


Semi-structured telephone interviews were conducted by the main researcher for the evaluation with 
eight DA coaches selected randomly from the 16 coaches. 


Teaching assistant/teacher survey 


Hard copies of the both teaching assistant/teacher and student surveys were sent to the schools with 
postage-paid, self-addressed return envelopes. The teaching assistants were to return the student 
surveys along with their own completed survey(s). Data from these surveys were used to measure 
implementation of fidelity. Both teaching assistant and student surveys were anonymous. 


Towards the end of the implementation period, all the teachers/teaching assistants who implemented 
the UofS programme in all the schools were asked to complete a survey on their perceptions of the 
programme and its implementation, and its impact on children’s reading and spelling. It contained 19 
questions, some Likert-scale and some open-ended. Thirty-five out of 61 teaching assistants 
completed the questionnaire. 


The teaching assistant survey data was summarised by summing the scores of the numerical data 
and looking for patterns in the responses to the open-ended questions. 


Student survey 


Towards the end of the implementation students in the UofS condition completed a brief survey on 
paper. The students responded to five questions about their perceptions of UofS. 


The first question showed a bar with a happy face at one end labelled “Like a lot” and a sad face at 
the other end labelled “Do not like”. The students made a mark on the line that represented how they 
much they liked UofS. 


The remaining four questions were Yes/No responses with an option of a “?” if they were not sure 
how to respond. 


The questions were: 


e Units of Sound helps me read better. 

e Units of Sound helps me spell better. 

e Units of Sound helps me feel better about school work. 
e Units of Sound helps me feel better about myself. 


To analyse the first question, “Do you like Units of Sound?” the line was divided into 15 segments and 
a score from 1 to 15 allocated to the segment that was ticked by the student. 


Where possible, we used descriptive statistics to summarise the students’ responses in addition to 
looking for thematic patterns in the responses. 
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Impact evaluation 


Timeline 


The project took place over two phases, Phase 1 implementation began in spring 2013 and Phase 2 
in autumn 2013. See Table 1 for a detailed timeline. 


Table 1. Timeline 


Phase 1 


January Select and randomly assign students 


June — July Post-test 


Phase 2 


August — September 2013 — Recruit schools 
September Select and randomly assign students 


September Train school staff 


September — October Pre-test 


October — February Implement programme 
February — March Post-test 


July — October Analyse data 


October — November Write report 


Participants 


Dyslexia Action sought to recruit 50 secondary schools in areas with high levels of deprivation across 
England. Fifty was believed to be an adequate number of schools, even with substantial attrition, 
because randomisation was at the individual student level. 


Recruitment of this number of schools proved to be a challenge and the EEF allowed DA to run the 
project in two phases. In Phase 1 they signed up 24 schools. In Phase 2 they recruited 21 schools, 
making a total of 45 schools. 


There were a total of 446 students allocated in Phase 1 (223 students in each condition) and 340 in 
Phase 2 (170 students in each condition). See participant flow chart for the combined Phases 1 and 2 
sample in Figure 4 below. Flow charts for the separate phases can be found in Appendices 4 and 5. 


There was a 46% attrition rate, which really limits conclusions one can draw from the study. 
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Figure 4. Participant flow chart - Phases 1 and 2 combined 
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¢ Schools withdrew after pretest (s=1, p=10) 
« Schools did not posttest (s=5, p=50) 


+ Absent/Left (p=26) 


Posttests completed (s=35, p=261) 


Excluded from analysis 
¢@ Schools with no pretests (s=2, p=29) 
¢ Individuals with no pretest (p=7) 


Analysed (s=33, p=225) 


within school 
(s=45, p*=786) 


Allocated to control (p=393) 


¢ Schools withdrew before pretest (s=4, p=45) 
¢ Individuals withdrew before pretest (p=2) 


Pretested (s=41, p=346) 


¢ Schools withdrew after pretest (s=1, p=10) 
+ Schools did not posttest (s=5, p=51) 

¢ Individuals withdrew after pretest (p=2) 

« Absent/Left (p=31) 


Posttests completed (s=35, p=252) 


Excluded from analysis 

¢ Schools with no pretests (s=2, p=30) 
¢ Individuals with no pretest (p=18) 

¢ Extreme outliers (p=2) 


Analysed (s=33, p=202) 


"s = number of schools 


*p = number of students 
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School characteristics 


The schools that were recruited to participate were in areas with high levels of deprivation and large 
numbers of students attracting the pupil premium. The recruited schools were predominantly urban 
academies, with Good or Requires Improvement Ofsted ratings. They were mostly low-achieving 
schools with 42% of students on FSM. See Tables 2a to 2d below for the characteristics of the 


schools. 


Table 2a 


School seiting Number of schools 


Randomised (s=45) Analysed (s=33) 


Town and fringe 2 1 
Urban > population of 10,000 43 32 


Table 2b 


School type Number of schools 
Randomised (s=45) Analysed (s=33) 
Academies 19 


LEA maintained schools 


Table 2c 
On (Yom e-1ilare] Number of schools 
le teTae (oyun tx-10 Mk (=f—7- be) Analysed (s=33) 
Good 


Requires improvement 14 8 
Inadequate 7 6 


Table 2d 
IN TUT a0} f=) ake) maxed pLeye) [} 


Randomised Analysed 
(s=45) (CER) 


Percentage achieving five grade A*-C GCSEs 45 50 
including English and Maths 


Percentage eligible for free school meals 42 31 
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Pupil characteristics 


There was a much higher percentage of boys in the study than girls and a higher percentage of 
students on free school meals. See Tables 2e and 2f for the pupil-level characteristics. The data for 
most of the children who dropped out are not available. 


Table 2e 


fer=vate(-1a Analysed sample 


Total Tahcclarceyanicey al (eveyaline) | 
Male 238 (56%) 136 (60%) 102 (51%) 
Female 160 (37%) 74 (383%) 86 (43%) 


'22 students not matched with NPD (5%) and seven where gender not provided by NPD (2%) 


Table 2f 
Free school meals” Analysed sample 


Total Tahtclaczialicey al (eveyaline) | 


238 (56%) 132 (59%) 106 (53%) 


167 (39%) 83 (37%) 84 (42%) 


?22 students not matched with NPD (5%) 


Attrition 


As the participant flow chart illustrates, DA recruited 45 schools before randomisation. Four schools 
withdrew after randomisation but before pre-testing and another school withdrew after pre-testing. 
Reasons for attrition included schools’ inability to commit to the staff training, overlapping 
interventions, technology, staffing and time-tabling issues. The early withdrawals seemed to occur 
because head teachers had not understood the requirements of implementing the programme, 
including the extensive initial online training. The DA staff tried to keep the schools from dropping out 
but additional staff changes and challenges of scheduling the UofS sessions and/or room allocations 
made it impossible to keep them in the study. 


Five schools that did not officially withdraw from the study did not administer the post-tests. This left 
35 schools but two of those had not administered the pre-test, so that left 33 schools and 427 
students in the analyses of the NGRT. As we were not sure whether the missing subjects were similar 
to the subjects remaining in the study, we decided not to impute missing values so this resulted in a 
45.7% attrition rate. 


Outcomes and analysis 


The pre-test analyses showed no significant difference between groups. Therefore, we conducted 
linear regressions on the NGRT and SWST post-test scores, using the NGRT pre-test scores as the 
covariate with treatment condition as the independent variable and phase as a fixed factor and school 
as a random factor. We found no significant differences between treatment and control student 
outcomes on either the post NGRT or the SWST, for the whole sample or for the students eligible for 
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FSM. We used Hedges’ g to calculate effect sizes on the NGRT and SWST outcomes. See Table 3 
for the results for the whole sample and of students eligible for FSM with data we received from the 
NPD. See Appendix 6 for all analyses including those for the separate phases. 


Table 3. Results for the overall and FSM samples 


Sig. Effect Confidence 


(OT ] Kore) pa=) fore) ave liareya) Number Post-test Post-test 
eNVZ=) | size intervals 


2 adjusted adjusted 


Student mean sd o\"(=1 aa] ©) oY = 18 


Units of 256.39 
Sound 

Control 262.59 
Units of 

Sound 

Control 

Units of 

Sound 

Control 

Units of 

Sound 

Control 


Note: None of these findings were statistically significant, suggesting that any differences occurred by chance. 


Fourteen schools received the programme from teaching assistants who had been trained and 
supported in UofS via distance learning only. Ten of these schools provided sufficient data to be 
included in the NGRT analyses. Analyses, controlling for pre-test reading scores, compared students’ 
post-test scores from schools in the face-to-face support condition, to those in distance learning only 
and to those in the control group. See Table 4 for the results, which show no significant differences 
and all negative effects. In the Distance/Face-to-face comparisons, the effects sizes favour the Face- 
to-face condition, although not significantly. 
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Table 4. Results for the comparison of distance-only versus face-to-face support 


Post- 
Number test Post-test 
Outcome Condition of adjuste adjusted 
Students d sd 
mean 


Distance 246.626 
Control 263.066 
Face-to-face 260.708 
Control 263.066 
Distance 246.626 
Face-to-face 260.708 
Distance 


lexeyayice(=yarex=) 
Sig. Effect Intervals 
ENT) size 


Lower Upper 


Control 
Face-to-face 
Control 


Distance 11.338 13.23 
Face-to-face 134 14.067 14.12 


Notes: None of these findings were statistically significant, suggesting that any differences occurred by chance. 


Cost 


The costs are reported per school. According to Dyslexia Action the costs covered by the EEF 
included: 


e An unlimited licence for the Units of Sound at £1,200 per school and £90 per school for the 
books that go with the programme. 

e The online training was £100 per person; on average two people per school were trained, 
therefore the cost was £200 per school. 

e DA provided headphones for the schools and four days of follow-up support and coaching for 
schools in the study. 


DA estimated that resources and training costs were about £2,500 per school purchasing the UofS 
programme. This cost would enable a school to teach as many students using UofS as they wished 
provided there were enough staff available to implement the programme. Salaries for teaching 
assistants were not included in the cost as they were already on the school staff and would simply be 
employed differently. If a school were to serve 10 students like they did in the study that would be 
£250 per student. 
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Process evaluation 


Implementation 


In both phases some schools encountered difficulties with technology related to the initial online 
reading and spelling tests, which delayed the start of the teaching schedule. Some schools could not 
access the tests and others could not get the tests to run on their computer networks. There were 
also difficulties in downloading the UofS software in Phase 1. These were resolved in Phase 2 and 
downloads were completed and checked before the start of the autumn term. 


Two members of staff from each school (usually a teacher and a teaching assistant but sometimes 
two teaching assistants) were identified by the school administration to implement the programme. 
There were a total of 115 staff members who were enrolled on the online UofS Practitioner's course, 
but time pressures meant that not all school staff had completed the training before the DA staff 
began work in the schools. In fact some of them did not do any of the online training. 


In Phase 1, there were a total of 54 members of staff. Twenty-two members of school staff completed 
the ten modules of the online UofS Practitioner’s course, 18 completed some of the modules, and 14 
did not access the course at all. For Phase 2, DA and evaluation staff stressed the importance of the 
online training and set a deadline for completion. There were 61 staff members; however, only 17 
completed all ten modules and 23 did not access the course at all. We do not know why each person 
did or did not complete the training so we could not analyse their students’ outcomes. We heard from 
the interviews that there were technical problems in some schools and shortage of time for some 
staff; some perceived the modules as too time-consuming. 


Fidelity 


We do not know how many sessions students actually engaged in; however, there were many 
circumstances that interfered with a high level of implementation fidelity. The intervention was 
delivered as intended to very few schools. Here were the main issues: 


e Some teachers were late in completing the online training or did not complete it at all. 

e In many cases, the implementation took place for fewer than 17 weeks, often owing to delays 
in administering the placement test. Schools were reminded frequently by the DA staff to 
administer the placement tests. 

e There were school-based technical problems with accessing both the online tests and the 
UofS programme. 

e There was inconsistent support by the DA coaches, often because schools would not allow 
them to schedule visits. 

e There were school-based organisational problems such as staffing, room availability, and 
timetabling. 

e Despite clear lines of communication set up between school staff, DA, and Testwise (the 
group running the online tests), schools often struggled to communicate with the appropriate 
organisation to resolve their particular problems, such as accessing the training, running the 
programme, or administering the online assessments. 

e The programme demanded a high level of commitment and time from the school staff 
involved in the project. There was a lot of information to access and absorb, plus at least 20 
hours of training and support. In many cases this was hard to achieve in the time allotted. 


Despite these issues, the intervention was rated positively by the teaching assistants and the 
students; most respondents believed that the intervention improved the students’ reading abilities. 
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Process evaluation outcomes 


Analysis of the DA coach interviews 


Seven of the eight coaches interviewed each supported one school in implementing the programme; 
the other coach supported two schools. A content analysis of the interviews was conducted. 


The coaches reported overall positive perceptions on the part of the school staff and students of the 
UofS programme. However, there were recurrent issues that the coaches reported in supporting the 
implementation of many of the schools. These included: 


1. Technical issues with implementing the programme, most often at the beginning of the 
implementation. These seemed to result from a lack of technical support in the school for the 
project. Sometimes the UofS programme did not run at all initially or ran very slowly, resulting 
in frustration both for the teaching assistants/teachers implementing the programme and for 
the students trying to use it. Usually these problems were overcome with proactive support 
from the DA coaches and DA technical support staff. 

2. Teaching assistants and teachers who were implementing the programme did not all 
complete the online training before implementation. The online training was supposed to be 
completed before the DA coaches met with the staff. About half of the teaching 
assistants/teachers had not completed this training and in some cases no one in the school 
had done so. 

3. The lack of training issue was perceived to be the result of a lack of support by the head 
teacher and/or senior management for the project. This resulted in staff not being released to 
complete the training and also to timetabling issues of making staff available to facilitate the 
tutoring when the computers and students were available. 

4. Staff turnover was an issue in some schools, which led to cancelled sessions and 
replacement staff who lacked training. 


The coaches reported that the face-to-face training was well received by the school staffs. Coaches 
felt that the modelling that they provided was an important element in getting the level of good 
implementation fidelity in some schools. 


Summary of the teaching assistant/teacher survey 


The teaching assistant survey contained 19 questions, some were Likert-scale questions and others 
were open-ended. Unless otherwise indicated the Likert-scale questions were on a four-point scale 
with 1 being the least positive response and 4 the most positive. Thirty-five out of 61staff members 
completed some or all of the items. Staff members were reminded by the researchers numerous 
times. Nineteen of the staff members who completed the survey had participated in the online 
training. 


Below is a summary of their responses. For a detailed summary of the responses to each question, 
see Appendix 7. 


Barriers/perceived areas for improvement 


Technological problems were the most common barriers that were reported by the teaching 
assistants, particularly problems getting the UofS programme installed on the school’s computer 
network and accessing the NGRT. Some teaching assistants reported not being able to retrieve 
students’ work. 
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Perceptions of training, support and the UofS programme 


Training was mostly perceived to be supportive but a few thought that it was too time-consuming and 
perceived as too detailed. The two main strengths reported were that it was helpful to be able to try 
the programme out and to have someone there to answer questions as they arose. 


Some teaching assistants would have liked more experience with the programme before 
implementing it with the students. Most respondents appreciated the immediate feedback that they 
received in their implementations. 


A number of teaching assistants reported challenges with the online training (such as it was time 
consuming and/or they had technical problems with it). Some said that it was difficult to follow the 
training without being able to try out the programme at the same time. 


Most of the teaching assistants found the UofS programme useful and student engagement high, 
especially initially. A few reported low student interest and attendance but on average interest and 
attendance were high. Some challenges with the programme were technical difficulties, especially at 
the initial setup. Teaching assistants frequently stated that sound quality was an issue. 


Perceived outcomes 


On average teaching assistants responded that students made quite a lot of progress in reading 
ability as a result of the programme, and almost as much in spelling ability. Half of the respondents 
indicated that students gave mixed feedback and half reported positive feedback. 


Seven of the 12 respondents whose students had completed the final UofS placement tests at the 
time of the survey reported that their students progressed; two stated their students did not progress; 
and three stated some did and some did not. 


Formative findings 


In addition to resolving the technology, staffing and scheduling issues, 16 respondents made other 
suggestions for improving the UofS programme, such as adding games, providing more feedback on 
what students had not mastered, and combining the activities with comprehension work. 


Summary of student survey of Units of Sound 


One hundred and thirty-nine students completed the five-question paper-and-pencil survey, but not all 
answered every question. 


Q1: Do you like Units of Sound? 

Question 1 had a range of 1-15, with one indicating “do not like” and 15 indicating “like a lot”. 
139 responded; mean response was 10.90; 42 students (80%) indicated that they liked it a lot. 
Q2: Units of Sound helps me read better. 

139 responded; 112 students (81%) answered Yes. 

Q3: Units of Sound helps me spell better. 

139 responded; 112 students (81%) answered Yes. 


Q4: Units of Sound helps me feel better about school work. 
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138 responded; 87 students (63%) answered Yes. 
Q5: Units of Sound helps me feel better about myself. 
139 responded; 81 students (58%) answered Yes. 


Despite all the technological challenges and implementation issues with the UofS programme, most 
students rated the programme positively and believed that it helped them with their reading and 
spelling. However, only 139 students completed the surveys so the findings very well may not be 
representative of the 427 students who were included in the final analysis of the reading achievement. 


Alternative provision for control group 


We received little information about the provision for students in the control group. In most schools it 
seems there was none. In some schools there were interventions such as: 


e Read Write Inc 

e Rapid Plus, online-based reading support 

e reading clubs, which involved reading to a volunteer adult three times a week at lunchtime 

e Literacy Catch-up, which involved completing levelled worksheets 

e EAL lessons for EALs 

e Word Wasp or Hornet — a manual-based programme that teaches the rules and structures of 
English 


Limitations to the intervention 


The UofS programme demanded a high level of commitment and time from the school staff involved. 
There was a lot of information to access and absorb, plus at least 20 hours of online training and 
follow-up support. In many cases this was hard to achieve in the time allotted. 


There were many challenges that a number of schools had with the technology, particularly with 
getting the programme installed. Some of the technological issues identified in Phase 1 were 
corrected for Phase 2 but there were still a few schools who struggled with the technology. 


Some teachers were late in completing the online training or did not complete it at all, which probably 
reduced the implementation fidelity. In many cases, the implementation took place for fewer than 17 
weeks, often owing to delays in administering the placement test. 


There was inconsistent support by the DA coaches, often because they could not get into the schools, 
owing to the schools’ scheduling problems. There were other school-based organisational problems in 
implementing the programme, such as staffing, room availability, and timetabling. 
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Conclusion 


This attempt to evaluate the UofS programme was severely compromised and no firm conclusions 
can be drawn from it. There was no evidence of a significant positive impact of the UofS programme 
on the reading or spelling achievement of the struggling readers. Neither did it find significant 
differences in outcomes for the FSM students. However, although these outcomes are disappointing, 
the many problems with the evaluation mean that we cannot determine whether the programme is 
effective or not. 


Despite the lack of impact on the achievement outcomes, the responses of both the teaching 
assistants and student surveys were generally very positive. The teaching assistants believed that 
UofS improved the students’ reading and to a lesser extent their spelling. 


Limitations with the evaluation 


The evaluation started out with a strong design but challenges with recruitment and problems with 
schools dropping out mean that the evaluation did not provide a reliable assessment of the 
effectiveness of the programme. For example, the recruitment was spread out over such a long period 
that we had to assign the students to condition before the pre-test was administered. The researchers 
attempted to get the schools that dropped out of the study to complete the post-test but none of them 
did so. This severe limitation made it impossible to conduct intent-to-treat analyses so the analyses 
are only on the treated students. 


Even though continual efforts were made by the DA coaches and IEE researchers to get schools to 
have their students complete the online tests, there were many who did not, both at pre-test and at 
post-test. Despite clear lines of communication set up between school staff, DA and Testwise (the 
group running the online tests), schools often struggled to communicate with the appropriate 
organisation to resolve their particular problems (such as accessing the training, running the 
programme, or administering the online assessments). 


There was significant attrition during the study. Because the analyses were conducted at the 
individual level, the schools dropping out probably did not bias the internal validity of the findings, but 
it makes the generalisability of the findings to other schools impossible. 


Because so few of the students completed the survey, some because many schools did not 
administer it and some because the students were absent, these reported perceptions of the 
programme should be interpreted with caution. They may not be representative of all the students 
who experienced the programme. 


Because UofS was an online programme, any contamination of students in the control condition was 
unlikely. Students in the UofS condition may have talked about the UofS programme with their friends 
in the control group and increased their interest in reading, but this is not a serious risk to the validity 
of the results. Because the tests were administered online, we assume that there was no intervention 
in its administration by teaching assistants but we cannot be certain of that. 


Interpretation 


Based on preliminary evaluations of UofS and research on similar programmes, one would expect 
that UofS would have significant and at least moderate effects on students’ reading ability. 
Unfortunately, owing to the limitations of the intervention’s implementation and the evaluation, we 
think that this study was not a very secure evaluation of the UofS programme. Recruitment, 
technology, attrition, implementation and testing issues all contributed to an evaluation whose findings 
need to be interpreted with extreme caution. 
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Future research and publications 


Some of the suggestions below would also be applicable to the general implementations of the UofS 
programme. 


Future research should allow a longer time for recruitment and involve all stakeholders in the process. 
The EEF could consider funding a longer start-up time for recruiters to conduct face-to-face meetings 
with potential schools, including a presentation by the evaluators on RCTs so that potential 
participants understand the process and thus hopefully reduce attrition. 


Teachers do not have a lot of spare time. It might help to have one point of contact for the schools so 
they know who to go through and are not bothered by being contacted by numerous people. DA, the 
researchers and Testwise tried to do that but it did not always work out. 


Owing to the challenges that a number of schools had with the technology, it would be wise to ensure 
that schools had sufficient technical support before agreeing to an implementation at a school, or 
have the provider provide support to improve implementation fidelity. Another option would be for DA 
to provide that support, but that would increase the cost of the programme. 


It might be better if schools had a preparatory year to learn the programme, followed by an 
experimental year. Although it is difficult to get schools to commit to having students in the control 
group for a long period, they could tutor students in one Year level in the preparatory year and then 
do the study with a subsequent cohort the following year. 


Perhaps requiring schools to pay for some of the costs might increase the commitment of schools to 
engage with the project and reduce attrition. 


We plan to submit a paper to a peer-reviewed journal based on the evaluation for publication after 
EEF releases the final report. We will summarise the effects of the programme along with other 
evaluations of UofS and add the information to our Evidence4impact website. We will report the 
findings in Best Evidence in Brief. 
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Appendix 1: Head teacher letter of agreement 


Evaluation of Dyslexia in Action Units of Sound, 2012-13 


Units of Sound (UofS) is a highly structured computer-based literacy scheme that has great potential 
to improve the engagement and reading attainment of struggling readers. It is recommended that one 
hour per week is timetabled for students to work independently on the programme, in groups of five, 
supported by a teacher who will monitor its use and do a weekly check-reading exercise to ensure 
that the student is working at the appropriate level. Each student should cover 2 pages for reading, 2 
pages for spelling, 1 set of UofS exercises, 1 check-reading with the teacher, and also possibly either 
a memory or dictation activity. An additional 30 minutes session for independent UofS work is 
recommended. UofS provides instant feedback to the students and highlights to the teacher what 
content students have mastered and where students are struggling. Individual students’ progress 
through the programme can be tracked through printed records of work from each UofS session. 


This school year, researchers at the Institute for Effective Education (IEE) will conduct a study to 
assess whether UofS improves children’s literacy and reading ability in particular. The study will focus 
on the implementation of UofS with Year 7 students in the Spring and Summer terms. 


Schools will be invited to participate that are in areas of high multiple deprivation with students who 
attract the pupil premium. Eligible student participants will be those who achieved below Level 4 in 
English in the Key Stage 2 SATs and who are eligible for Free School Meals. 


Eligible students will complete a pre-test before the implementation of UofS starts. On the basis of the 
pre-test results, researchers at the IEE will select a maximum of 20 and a minimum of 10 students at 
each school to take part in the evaluation in the school year 2012-13. The researchers will then 
randomly assign the students to work with UofS (the intervention group), or not to work with UofS (the 
control group) but to receive the intervention in September 2013 as a delayed treatment. Towards the 
end of the Summer term, students in both the intervention and control groups will complete a post-test 
to provide data of programme effectiveness. 


Online training plus face-to-face support will be provided by expert Dyslexia Action teacher-coaches 
to two teachers/staff members in each of 40 schools who will deliver the programme. In an additional 
ten schools, only online training and support will be provided. Schools will be randomly assigned to 
receive online plus face-to-face training and support, or online only training and support. This will 
permit a subsidiary analysis of the effectiveness of contrasting training modes. 


At the end of the study, teachers and students will be surveyed online about their experiences and 
perceptions of UofS, and telephone interviews will be conducted with sub-samples of DA teacher- 
coaches. The study will provide valuable information about the effectiveness of the programme for 
improving literacy and reading in particular. It will also provide insights into the quality of resources 
and how teachers use them, and the effectiveness of different training approaches. 


Timeline for research project, 2012-13 


Autumn Term 


Dyslexia Action (DA) recruits schools 

The IEE randomly assigns schools to online plus face-to-face training or to online only training 
Schools nominate two staff members to participate 

Schools and teaching staff are notified of the time-line for training, pre-testing, timetabling and 
logistical arrangements for implementation to begin by January 14, 2013 

e Schools identify potential student participants (<Level 4 KS2 SATS (English) and eligible for 
Free School Meals) 
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e Prospective student participants are screened using the UofS screening test 
e EE researchers randomly select between 10 and 20 students from each school to participate, 
aiming for 20 where possible 


e DA matches teacher-coaches to the participating schools 
e DA teacher-coaches prepare training materials and make these available to the schools 


e DA technical support staff ensure that schools have the technical equipment required for 
training and implementation 


Spring Term 


e January 7-14: Students complete the pretest 
e By January 14: Teachers complete the online training 


e Week beginning January 14: Implementation of UofS starts with students in the intervention 
group and continues throughout the term 


e Weekly individual students’ UofS progress records are provided to the IEE 


Summer Term 


e Implementation of UofS continues throughout the term with students in the intervention group 
e Weekly individual students’ UofS progress records are provided to the IEE 

e Towards the end of term, students complete the post-test 

e Towards the end of term, teaching assistants and students complete surveys 


e Towards the end of term, a sub-sample of telephone interviews conducted with DA teacher- 
coaches 


Please complete and return the form below if you agree for your school to take part in the UofS study 
and accept the eligibility terms and conditions for receiving the software and training. Please note that 
a swift reply would be welcome. We hope you will decide to take part in this exciting research study. 


Regards, 


len 


Project Manager 
Telephone: 01904 328159 


Seac_aw 


Email: mary.sheard@york.ac.uk 


Fax: 01904 328156 
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Evaluation of Dyslexia in Action Units of Sound, 2012-13 


Eligibility Terms and Conditions for participation 


e Nominated teachers undertake and complete the training as required by January 14, 2013. 


e Timetables are prepared to accommodate the required number/length of UofS sessions with 
the participating students throughout the Spring and Summer terms. 


e Teachers will oversee the completion of the pre- and post-tests for the participating students 
within the timeframe and conditions specified by the IEE. 


e Schools will ensure that the technology to be used in the UofS sessions is of high enough 
quality that students can work efficiently and effectively with the programme. 


e Schools will provide access to technical support staff in order that the software and equipment 
can be installed and used throughout the research study. Where possible, technical support 
staff will attend the training sessions. 


e Teachers will, at the earliest opportunity, notify DA and the IEE if there are any support or 
operational issues preventing the effective use of the software or equipment. 


e The named UofS co-ordinator in each school will provide the IEE with a regular weekly 
individual students’ progress report in the form of UofS records, by fax or mail. 


e Teachers and students will complete surveys about aspects of UofS, provided by the IEE. 

e Teachers will provide valid email addresses and telephone contact numbers to the 
researchers and DA teacher-coaches, and will agree to check communications regularly 
during the period of the research study. 


e The school should not simultaneously be participating in another EEF programme evaluation. 


Please complete and return the form to the address below if you agree to take part in the UofS 
study and accept the eligibility terms and conditions. 


Please note that application is not a guarantee of participation and IEE will confirm 
acceptance. 
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Evaluation of Dyslexia in Action Units of Sound, 2012-13 


Head teacher agreement 


| agree for my school to take part in the Units of 
Sound study and | accept the eligibility terms and conditions. 


Name of Head Teacher: 


Signature of Head Teacher: 


Email address 


School Address 


School email 


School FAX 


School Telephone Number 


% Free school meals at school % EAL Students at school 


Please return by post to Mary Sheard at the address below, or by FAX on 01904 328156. 


Email: mary.sheard@york.ac.uk 


Education Endowment Foundation 31 


Units of Sound 


Appendix 2: Letter of consent for parents 


Evaluation of Dyslexia Action Programme to Teach Reading and Spelling 
Dear Parent/Guardian, 


We would like to request your permission for your child to take part in an educational research study. 
The following information explains why the research is being done and what it would involve for your 
child. 


What is the Institute for Effective Education? 


The Institute for Effective Education (IEE) is part of the University of York. It aims to find out what 
works in teaching and learning and why, and then use the evidence to improve education. 


What is the Units of Sound programme to teach reading and spelling? 


Units of Sound is a second chance' phonics based Reading and Spelling computer programme 
suitable for students of all ages who need support with their literacy. With a screening placement, 
student progress display and a screen tutor to show students how to use the programme correctly, 
Units of Sound enables all students to learn independently. 


What is the purpose of this study? 


This study is being done to see if the Units of Sound programme helps students to achieve better in 
reading and spelling. 


Why is my child’s class participating? 


We will conduct this study with Year 7 students in approximately 50 secondary schools in England. 
The head teacher of your child’s school agreed to participate in this study. 


In each school, approximately 20 students who scored below Level 4 in reading at the Key Stage 2 
SATs will be randomly selected to take part in the study. Half of the randomly selected students will 
then be randomly assigned to work with the Units of Sound programme as the intervention group, and 
the other half will be the control group who will not work with the programme during the evaluation 
period. 


What will happen in the study? 


In January 2013, the students in both the intervention and control groups will do an online reading 
test, which will be repeated in June 2013, along with an online spelling test. After the testing in June, 
schools will be able to use Units of Sound with students as they wish. 


Does my child have to take part? 


By taking part, your child will receive additional support to improve reading and spelling. However, 
you may choose not to permit your child’s test scores to be used in the research study. If this is the 
case, please complete and sign the attached opt-out form by 3:00 pm on (one week later). A 
student’s right to withdraw from the testing will be respected. 


What will my child’s participation be? 
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Your child will take the online reading test in January 2013 and at the end of the study in June 2013, 
along with an additional online spelling test. If assigned to use Units of Sound, your child will work 
independently on the Units of Sound programme for one hour per week in a small group supported by 
a member of staff, with another thirty minute session working individually. Sessions will be conducted 
in quiet work areas where students will feel comfortable to work on the programme. This will start in 
January 2013 and finish at the end of June 2013. 


What should | tell my child about the study? 


It would be helpful if you could tell your child that the research study is trying to find out whether using 
Units of Sound helps students to read and spell better. 


What are the disadvantages and risks of taking part? 


There are no known disadvantages or risks in working with the Units of Sound programme, which has 
been available and used in schools for many years. Students are placed on the programme at a level 
matched to their ability and needs. The staff supporting the programme will have undertaken detailed 
training in how to place students appropriately on the programme, ensure that the programme is used 
effectively, and to check students’ individual progress in reading at each session. 


The sessions will be timetabled, so the students will possibly be taken off their regular timetable for up 
to ninety minutes per week. 


What are the possible benefits of taking part? 


By participating in this study your child will experience a new strategy for improving reading and 
spelling alongside the usual approaches used in your child’s school. The information gained from this 
study may influence how your child and others will be supported in reading and spelling in the future, 
as the Units of Sound programme is developed to meet the needs of students throughout the 
secondary school age range. 


What happens when the research stops? 


IEE researchers will analyse the test scores to determine the overall effectiveness of the Units of 
Sound programme. Scores for individual students will be kept confidential. 


When the research is over, the school will receive a report that will show if using Units of Sound 
makes a difference to students’ achievement in reading and spelling. The school can then decide 
whether and how best to use Units of Sound in school. 


Will my child’s information be kept confidential? 


Yes. Student data will be communicated to the researchers in a secure manner, and student names 
will be replaced with code numbers. No identifiable student’s data will appear in any report. 


What do | do if | have a concern? 


If you have a concern or question about your child’s participation in this study, please contact Mary 
Sheard (e-mail: mary.sheard@york.ac.uk) Tel: 01904 328159 or Bette Chambers (e-mail: 
bette.chambers@york.ac.uk) Tel: 01904 328153 
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Parent/Guardian opt-out form 


If you do not permit your child’s test scores to be used in the study, please complete this form and 
return it to your child’s form teacher by 3:00 pm on (one week after receipt). 


| do not wish my child’s test scores to be used in the research project. 


Student’s name: 


(Please print clearly) 


School: 


Form Teacher’s Name: 


Parent’s/Guardian’s name: 


(Please print clearly) 


Parent’s/Guardian’s signature: 


Date: 
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Appendix 3: Information and consent form for Dyslexia 
Action coaches 
Evaluation of Dyslexia Action Programme to Teach Reading and Spelling 


This school year, researchers at the Institute for Effective Education (IEE) will conduct a study to 
assess whether UofS improves children’s literacy and reading ability in particular. The study will focus 
on the implementation of UofS with Year 7 students in the Spring and Summer terms, 2013. 


Schools will be invited to participate that are in areas of high multiple deprivation with students who 
attract the pupil premium. Eligible student participants will be those who achieved below Level 4 in 
English (Reading) in the Key Stage 2 SATs and who are eligible for Free School Meals. 


Researchers at the IEE will select a maximum of 20 and a minimum of ten eligible students at each 
school to take part in the evaluation in the school year 2012-13. The researchers will then randomly 
assign the students to work with UofS (the intervention group), or not to work with UofS (the control 
group) but to receive the intervention in September 2013 as a delayed treatment. Eligible students will 
complete a pretest before the implementation of UofS starts. Towards the end of the Summer term, 
students in both the intervention and control groups will complete post-tests to provide data on 
programme effectiveness. 


Online training plus face-to-face support will be provided by expert Dyslexia Action teacher-coaches 
to two teachers/staff members in each of 40 schools who will deliver the programme. In an additional 
ten schools, only online training and support will be provided. Schools will be randomly assigned to 
receive online plus face-to-face training and support, or online only training and support. This will 
permit a subsidiary analysis of the effectiveness of the contrasting training modes. 


At the end of the study, teachers and students will be surveyed about their experiences and 
perceptions of UofS, and telephone interviews will be conducted with DA teacher-coaches. The study 
will provide valuable information about the effectiveness of the programme for improving literacy and 
reading in particular. It will also provide insights into the quality of resources and how teachers use 
them, and the effectiveness of the different training approaches used. Please note that as a DA 
teacher-coach, you will be given the opportunity to check and comment on the analysis of the 
telephone interview(s) conducted with you as part of the data collection process. 


Please complete and return the consent form below to indicate if you agree to take part in the UofS 
evaluation and for anonymised research findings from interviews with you to be included in the final 
report and possible future academic publications and presentations. 
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Evaluation of Dyslexia Action Units of Sound, 2012-13 
Dyslexia Action Teacher-Coach agreement 


| agree to take part in the Units of Sound evaluation and for anonymised research findings from 
interviews with myself to be included in the final report and possible future academic publications and 
presentations. 


Name of Teacher-Coach: 
Signature of Teacher-Coach: 
Email address: 


Work Address: 


Work email if different from above: 


FAX: 


Work Telephone Number: 


Please return by email to mary.sheard@york.ac.uk 

or post to: 

Mary Sheard, Institute for Effective Education, Berrick Saul Building, University of York, 
Heslington, York, YO10 5DD 


or by FAX on 01904 328156 
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Appendix 4: Participant flow chart for Phase 1 


Excluded schools 
¢ Declinedto participate (s=64) 
¢ withdrew before randomisation (s=1) 


Individual randomisation 


within school 
(S=24, p*=446) 


Allocatedto intervention (p=223) Allocatedto control (p=223) 


+ Schools withdrew before pretest (s=3, p=40) 


¢ Schools withdrew before pretest (s=3, p=40) 


Pretested (s=21, p=183) 


Pretested (s=21, p=183) 


¢ Schools withdrew after pretest (s=1, p=10) ools withdrew after pretest (s=1, p=10) 
+ Schoold did not posttest (s=2, p=26) + Schools did not posttest (s=2, p=26) 
¢ Absent/Left (p=16) « Absent/Left (p=18) 


Posttests completed (s=18, p=131) Posttests completed (s=18, p=129) 


Excluded from analysis Excluded from analysis 
¢ Individuals with no pretest (p=3) ¢ Individuals with no pretest (p=2) 
« Extreme outlier (p=1) 


Analysed (s=18, p=128) Analysed (p=126) 
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Appendix 5: Participant flow chart for Phase 2 


Assessedfor eligibility(s'=120) 


Excluded schools 

¢ Did not meetinclusion criteria (s=1) 
¢ Declinedto participate (s=96) 

¢ Withdrew before randomisation (s=2) 


Individual randomisation 


Allocated to intervention (p=170) 


+ Schools withdrew before pretest (s=1, p=6) 


Pretested (s=20, p=164) 


+ Schools did not pasttest (s=3, p=24) 
¢ Absent/Left (p=10) 


Posttested (s=17, p=130) 


Excluded from analysis 
¢ Schools with no pretests (s=2, p=29) 
¢ Individuals with no pretest (p=4) 


Analysed (s=15, p=97) 


Education Endowment Foundation 


within school 
(S=21, p*=340) 


AlTocated to control (p=170) 


¢ Schools withdrew before pretest (s=1, p=5) 
¢ Individuals withdrew before pretest (p=2) 


Pretested (s=20, p=163) 


¢ Schools did not posttest (s=3, p=25) 
¢ Individuals withdrew after pretest (p=2) 
¢ Absent/Left (p=13) 


Posttested (s=17, p=123) 


Excluded from analysis 

¢ Schools with no pretests (s=2, p=30) 
¢ Individuals with no pretest (p=16) 

« Extreme outlier (p=1) 


Analysed (s=15, p=76) 
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Appendix 6: Table of all analyses completed 


Post-test Post-test Confidence 


of : ; Sig. Effect Intervals 
Phase Outcome Condition N adjusted adjusted ieval ice 


mean sd Lower Upper 


UofS 128 256.58 69.52 
Control 126 259.03 69.75 
UofS 97 254.94 89.27 
Control 76 267.58 82.25 
UofS 225 256.39 78.36 
Control 202 262.59 77.03 
UofS 98 13.95 14.77 
Control 102 14.94 14.90 
UofS 99 12.42 14.29 
Control 82 13.56 13.67 
UofS 197 13.19 14.50 
Control 184 14.25 14.30 
UofS 132 251.79 63.34 
Control 106 264.89 63.67 
UofS 113 12.42 12.67 
Control 94 14.15 12.57 
Distance 66 246.626 70.10 
Control 202 263.066 79.07 
1a2 Neate —2ceteface 189 260.708 7751 og 76 0.03 -0.24 

Control 202 263.066 
Distance 66 246.626 
Face-to-face 159 260.708 

Distance 63 

Control 184 

1a2 swst —acetoface 184 14.067 1418 a s30 0.02 0.24 

Control 184 

Distance 63 


0.724 -0.04 -0.28 0.21 


0.067 -0.15 -0.45 0.15 


0.214 -0.08 -0.27 0.11 


0.496 -0.07 -0.34 0.21 


0.413  -0.08 -0.37 0.21 


0.293 -0.07 -0.27 0.13 


0.065 -0.21 -0.46 0.05 


0.221 -0.14 -0.41 0.14 


0.058 = -0.21 -0.49 0.07 


0.150 -0.19 -0.47 


0.076 = -0.21 -0.50 


0.154 -0.20 -0.50 


Face-to-face 134 


Note: None of these findings were statistically significant, suggesting that any differences occurred by 
chance. 
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Appendix 7: Summary of responses to individual TA 
survey questions 


The teaching assistant (TA) survey contained 19 questions; some were Likert-scale questions and 
some were open-ended. Unless otherwise indicated the Likert-scale questions were on a four-point 
scale with 1 being the least positive response and 4 the most positive. Thirty-five out of a possible 61 
TAs completed some or all of the items. Below is a summary of their responses. 


Q1: How supportive was the face-to-face apprenticeship training? 


Twenty-nine of the 35 TAs responded to this question. Mean score was 3.24 and the mode was 4; 16 
TAs indicated that the training was perceived to be very supportive. 


Q2: What was the most useful part of the face-to-face training? 


There were 27 responses to this question, with two main strengths reported: 


e It was helpful to be able to try the programme out 
e It was helpful to have someone there to whom they could ask questions as they 
arose. 


They all said they felt comfortable asking questions. Typical comments were “Being able to discuss 
issues and receive immediate feedback and support”, “Actually having someone in the room who 
knew the programme inside out”. 


Q3: What was the most challenging part of the face-to-face training? 


The main challenges cited by most were technology issues: “It did not work properly at first. It was 
challenging to schedule convenient times for everyone.” Some participants had difficulty remembering 
the information and one mentioned that the trainer struggled to answer questions at times. 


Q4: Further comments about the training: 


Fifteen people responded to this question. The most common comments were positive ones about the 
trainers. 


Four TAs commented they would have benefited from more practice time without the students: “I think 
it would have been good to have spent time without the students. To be supported through the 
programme first.” 


Two TAs felt that they would have benefited from a video showing what they were supposed to do: 
“Instead of having a voice conference we would have benefited more if we had been shown a short 
film showing a practitioner working with a student or students Then we would have known what 
questions to ask during a voice conference.” 


Q5: Did you complete the online training? 

No =1, Yes =2 

Nineteen respondents replied Yes and 15 responded No. 
Q6: What was the most useful part of the online training? 


Twenty-six respondents replied with useful parts. The most often cited (seven respondents) was 
immediate feedback from assessment: “Immediate response to submitted units and example 
answers.” 


Also cited: becoming familiar with the programme’s parts; able to work at their own pace; the ability to 
speak with online mentor; and just the fact that it was online. 


There were two negative comments: one found it not useful; the other found it increasingly 
complicated and time-consuming as the units progressed. 


Q7: What was the most challenging part of the online training? 
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Thirty TAs responded to this question. The most-often cited problem was that the training was time- 
consuming (13 respondents). Two people could not complete it. Five people said they had technical 
problems during the training. Some stated that they had to be at work in order to do it. Many stated 
that the training was too detailed and technical. They said not all of the information needed was 
online. Two stated it was difficult doing the training without being able to try out the programme at the 
same time. 


° Two stated it was difficult when they wanted to go back and check something: “Losing all the 
correct answers when | returned to information to check something out.” 


. Several stated that the questions were not clear or relevant to what they had learned: “I felt 
some of the questions were ambiguous and not very clear’; “The questions asked didn’t seem useful 
at times because some of the information we felt hadn't been covered”; “The information at the start 
didn’t seem to bear any relationship to the questions at the end — it was very time-consuming, too 
much to do and sometimes impossible to find answers anywhere”. 


Q8: Further comments on the online training 


Seventeen people responded to this question. The time required was the main problem. Another 
common comment was that it was too detailed. “I think it could be helpful if adapted and simplified.” 


Four people stated that the help manual was useful. Three people suggested an external training prior 
to the computer training. 


Q9: What, if anything, went well in implementing the Units of Sound programme? 


Twenty-seven people responded to this question. 15 people stated that the students were engaged 
with the programme, especially initially: “The Students enjoyed all the computer based work & were 
able to work independently”; “The Students were quite enthusiastic, especially at the start. They like 
the fact it's on computers and it's at their level.” 


One respondent expressed a concern: “I found Units of Sound interesting but | believe students need 
assistance; otherwise they may rush so we don’t know how well they work.” 


Q10: What, if anything, was challenging in implementing the Units of Sound programme? 


Thirty-three respondents answered this question. Nineteen people stated they had technical 
difficulties, especially at the initial setup. They frequently stated that sound quality was an issue. “The 
internet is often playing up. The voice was not clear. Even the teachers struggled with recognising 
some of the words.” 


Two people stated the children were discouraged by their poor scores during baseline testing. “The 
initial test we found to be quite deflating for the students as they could see when an answer was 
marked negatively.” 


Low student interest was noted by five people. 

Attendance was a problem noted by five people. 

Q11: Was the programme delivered in the prescribed sequence? 
No=1, Yes =2 

Thirty-four TAs responded: 29 replied Yes and five replied No. 

Q12: Additional comments about implementing UofS: 

Fifteen people responded to this question. 


° Problems with the technology’s memory were cited by several. Students’ work was lost if they 
exited at the wrong point. Other students couldn’t progress past a certain point electronically. 


° One teacher stated the programme should start at the beginning of the school year; two 
stated it should start later in the year; and one liked it exactly as it is. 


. Again, sound quality problems were noted. 
Q13: How engaged were students with UofS? 


Thirty-four of the 35 TAs responded to this question. The mean score was 3.29 and the mode was 4, 
14 TAs indicating that the students were very engaged. 
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Q14: How good was student attendance at the UofS sessions? 


Thirty-four of the 35 TAs responded to this question. The mean score was 3.44 and the mode was 4, 
18 TAs indicating that attendance was very good. 


Q15: How much progress did students make in their reading ability as a result of their 
involvement in the UofS programme? 


Five-point scale: very little, little, some, quite a lot, a great deal. 


Twenty-five of the 35 TAs responded to this question. The mean score was 3.44 and the mode was 3. 
13 TAs indicated that students made quite a lot of progress in reading. 


Q16: How much progress did students make in their spelling ability as a result of the UofS 
programme? 


Five-point scale: very little, little, some, quite a lot, a great deal. 


Twenty-five of the 35 TAs responded to this question. Mean score was 3.08 and the mode was 3. 7 
TAs indicated that students made quite a lot of progress in spelling. 


Q17: What feedback did students give about the UofS programme 
no feedback, negative, mixed, positive 


Thirty-four of the 35 TAs responded to this question. The mean score was 3.38 and the mode was 3. 
16 TAs indicated that students gave mixed feedback and 16 gave positive feedback. 


Q18: Did the final placement tests confirm your view about the progress Students had (not) 
made as a result of the UofS programme? 


Twenty-five respondents answered this question. Most had not yet taken the test. Of the 12 that did, 
seven stated their students progressed, two stated their students did not, and 3 stated that some did 
and some did not. 


Q19: Add any additional comments you might like to make about the UofS evaluation. 


Sixteen respondents made additional comments. 


° Four people suggested adding games. 

. Two stated they would like more feedback on what the students had not mastered. 

. Three people suggested that the programme be used in conjunction with comprehension 
instruction/work. 

. Again, people mentioned the speech was difficult to understand (they had to switch between 


Northern and Southern accents) and the time constraints. 
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Appendix 8: Padlock rating 


1.Design: rn 3. Attrition: 4. Balance: =e 
What is the ee What is the Adjustment vallelty: 
quality of the aatectabie AV/-) re) mel ge) e) to rating to Be emare atcics 
design of the out from the account for : 
effect at the with 


evaluation? evaluation? balance? : : 
start? interpretation? 


Evaluation design Implementation Analysis and interpretation 


minimum Adjustment to 


ateilare| 1. Design 2. Power | 3. Attrition 4. Balance 5 Threats ic) 
(MDES) Wee tiny 


< 10% 


Fair and clear experimental 
design (RCT, RDD) < 0.3 < 20% 


Well-matched comparison 
(quasi-experiment) < 30% 


Matched comparison 
(quasi-experiment) 


Comparison group with 
poor or no matching 


No comparator 6 Imbalanced 


The final security rating for this trial is 1 @. This means that the conclusions have low security. 


The trial was designed as an efficacy trial and could achieve a maximum of 5@. This was a well- 
designed and powered trial, with a relatively small MDES of 0.14. However, substantial attrition 
(46%) has undermined the security of the findings. There were no other threats to the validity of the 
result. Therefore, the overall padlock rating is 1 @. 


Education Endowment Foundation 43 


Units of Sound 


Appendix 9: Cost rating 


Cost ratings are based on the approximate cost per pupil of implementing the intervention over one 
year. Cost ratings are awarded using the following criteria. 


Cost Description 


LEEEL High: up to £1,200 per pupil per year. 
LLLEEL Very high: over £1,200 per pupil per year. 
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